Repository navigation
feat(ts): honour a question's option_order across encoding and decoding - #1196
Conversation
|
Merged in 0.4.7. This is the port catching up on a field that reaches both halves of the pipeline, so a half-implementation would have been worse than none: the order places options in the prompt and the inverse maps the row back, and without the inverse the probabilities stay attached to the wrong options, silently. Three things I checked specifically. This also contains #1195's action-head fix, so the one conflict on the integration branch was that line. I resolved it in favour of this PR, which is the superset. |
feat(ts): honour a question's
option_orderacross encoding and decodingThe field is validated as a permutation of option indices, presented in the prompt in that
order, and the decoded row is put back into the caller's option order before an answer is
chosen — exactly how Python, Java, dotnet do it.
What:
checkQuestion: one slot per option, each option once; messages matchPython's
question %r: 'option_order' must be a permutation of range(%d)toInternalcarriesoptionOrderto the encoder (pass it intobuildQuestionPrefix)_decodeRowcallsunpermuteProbs(p, q.optionOrder)after softmaxingshortlist.ts: drop an order that no longer fits a narrowed label set (dropStaleOrder),emit the same warning text and call count as Python
How it was verified:
/tmp/probe_option_order.pycalls laya's ownAgent._decode_answerson these rows viaAgent.__new__with clamped temperatures,[0.25,-0.5]action, logits block width +3;tests/option-order.test.tsasserts those values for 8 rotated rows (choice/score/noul).tests/decode-parity.test.tsnow asserts all 47 cases including the 4option_orderarmsfrom
laya-java/fixtures/decode.json; 8 red at base → 0 on branch.agent/decoder) kill counts 17 new failures at base → 0 on branch; survivors=0.
laya-ts/tests/tsconfig.types.jsonprobe shows 0 diagnostics attributable to edits (the onlyTS errors are environmental: missing
@types/node/vitest).Why: without the un-permutation, the probabilities stay attached to the wrong options, which is
silent: the same logits read under different orders give different answers. The prompt half and
the decoder half have to move together.
Overlap with open PRs: deconflict hunk-level vs #340 (#1195 merged) — my edits touch
src/agent.tsaround line 787 where #340 addedanswerConfidence, andsrc/shortlist.tslines 281–294 where
predictTournamentnarrows choices; hunks are disjoint by line ranges.Checklist:
main(worktree detached ata0d27f37)